Average multi microbenchmarks results#5215
Conversation
…ks when creating suites
There was a problem hiding this comment.
Pull request overview
This PR updates the GC microbenchmark infrastructure to support aggregating (averaging) results across multiple microbenchmark runs/iterations, while also renaming/refactoring parts of the analysis/presentation pipeline and introducing an outlier-removal helper.
Changes:
- Add configurable microbenchmark iteration count (
iterations) and wire it into suite creation and execution. - Replace the previous single-result comparison flow with a new per-benchmark aggregation/comparison pipeline (
MicrobenchmarkResultComparison,GCTraceMetrics,GCTraceMetricComparisonResult). - Refactor output generation to primarily emit JSON (markdown generation currently disabled).
Reviewed changes
Copilot reviewed 21 out of 21 changed files in this pull request and generated 18 comments.
Show a summary per file
| File | Description |
|---|---|
| src/benchmarks/gc/GC.Infrastructure/GC.Infrastructure/Commands/RunCommand/CreateSuiteCommand.cs | Reads configured iteration count and applies it to microbenchmark suite environment. |
| src/benchmarks/gc/GC.Infrastructure/GC.Infrastructure/Commands/RunCommand/BaseSuite/MicrobenchmarksToRun.txt | Updates baseline suite benchmark list. |
| src/benchmarks/gc/GC.Infrastructure/GC.Infrastructure/Commands/RunCommand/BaseSuite/Microbenchmarks.yaml | Renames environment iteration setting to iterations. |
| src/benchmarks/gc/GC.Infrastructure/GC.Infrastructure/Commands/Microbenchmark/MicrobenchmarkCommand.cs | Runs microbenchmarks for iterations and switches to new aggregation/comparison logic before presenting results. |
| src/benchmarks/gc/GC.Infrastructure/GC.Infrastructure/Commands/Microbenchmark/MicrobenchmarkAnalyzeCommand.cs | Updates analysis-only command to use the new aggregation/comparison logic. |
| src/benchmarks/gc/GC.Infrastructure/GC.Infrastructure.Core/Presentation/Microbenchmarks/Presentation.cs | Changes presentation API to accept precomputed grouped results; markdown output path currently disabled. |
| src/benchmarks/gc/GC.Infrastructure/GC.Infrastructure.Core/Presentation/Microbenchmarks/Markdown.cs | Markdown generation code is commented out. |
| src/benchmarks/gc/GC.Infrastructure/GC.Infrastructure.Core/Presentation/Microbenchmarks/Json/JsonOutput.cs | Removes unused placeholder type. |
| src/benchmarks/gc/GC.Infrastructure/GC.Infrastructure.Core/Presentation/Microbenchmarks/Json.cs | Moves JSON generator to Microbenchmarks presentation namespace and updates signature for grouped results. |
| src/benchmarks/gc/GC.Infrastructure/GC.Infrastructure.Core/Configurations/Microbenchmarks.Configuration.cs | Renames iteration to iterations in microbenchmark environment configuration. |
| src/benchmarks/gc/GC.Infrastructure/GC.Infrastructure.Core/Configurations/InputConfiguration.cs | Adds iterations map to input configuration. |
| src/benchmarks/gc/GC.Infrastructure/GC.Infrastructure.Core/Analysis/Microbenchmarks/MicrobenchmarkResultsAnalyzer.cs | Removes old analyzer/comparison pipeline. |
| src/benchmarks/gc/GC.Infrastructure/GC.Infrastructure.Core/Analysis/Microbenchmarks/MicrobenchmarkResultComparison.cs | Adds new JSON/trace mapping, per-benchmark analysis, and aggregation/grouping logic. |
| src/benchmarks/gc/GC.Infrastructure/GC.Infrastructure.Core/Analysis/Microbenchmarks/MicrobenchmarkResult.cs | Introduces new MicrobenchmarkResult model (namespace currently mismatched vs usage). |
| src/benchmarks/gc/GC.Infrastructure/GC.Infrastructure.Core/Analysis/Microbenchmarks/MicrobenchmarkComparisonResult.cs | Updates comparison to support averaged values/outlier removal and new trace-metric comparisons. |
| src/benchmarks/gc/GC.Infrastructure/GC.Infrastructure.Core/Analysis/GCTraceMetrics.cs | Adds trace-derived metric extraction (includes reflection/stat bugs). |
| src/benchmarks/gc/GC.Infrastructure/GC.Infrastructure.Core/Analysis/GCTraceMetricComparisonResult.cs | Adds averaged comparison for trace metrics (baseline vs comparand). |
| src/benchmarks/gc/GC.Infrastructure/GC.Infrastructure.Core/Analysis/GCTraceMetricComparison.cs | Adds helper wrapper for metric comparison construction. |
| src/benchmarks/gc/GC.Infrastructure/GC.Infrastructure.Core/Analysis/BdnJsonResult.cs | Refactors BDN JSON model types; renames top-level to BdnJsonResult. |
| src/benchmarks/gc/GC.Infrastructure/GC.Analysis.API/Statistics.cs | Adds RemoveOutliers helper (IQR method). |
| src/benchmarks/gc/GC.Infrastructure/Configurations/Run.yaml | Adds iteration configuration block (currently mismatched with new iterations input model). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
…chmarks namespace
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 21 out of 21 changed files in this pull request and generated 8 comments.
Comments suppressed due to low confidence (1)
src/benchmarks/gc/GC.Infrastructure/GC.Infrastructure.Core/Presentation/Microbenchmarks/Json.cs:15
Json.Generatedoesn’t use theconfigurationparameter, and theusing GC.Analysis.API;/using GC.Infrastructure.Core.Presentation.GCPerfSim;directives are unused. Consider removing the unused parameter/usings to avoid warnings and keep the API surface minimal.
…for each projection
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 22 out of 22 changed files in this pull request and generated 3 comments.
Comments suppressed due to low confidence (2)
src/benchmarks/gc/GC.Infrastructure/GC.Infrastructure.Core/Analysis/Microbenchmarks/MicrobenchmarkResultComparison.cs:155
- When trace collection is enabled,
tracePathfalls back to an empty string if the JSON path isn’t present injsonToTraceMap, and that empty path is passed toAnalyzerManager.GetAnalyzer(...), which will fail with a less actionable error. Prefer validating the mapping result (e.g.,TryGetValue) and throwing an exception that includes the missing JSON path / benchmark name, or ensuringMapJsonToTraceguarantees coverage for everyjsonPath.
if ((!excludeTraces) && configuration.TraceConfigurations.Type != "none")
{
string outputPathForRun = Path.Combine(configuration.Output.Path, run.Name!);
string tracePath = jsonToTraceMap.GetValueOrDefault(jsonPath, "");
using (var analyzer = AnalyzerManager.GetAnalyzer(tracePath))
{
src/benchmarks/gc/GC.Infrastructure/GC.Infrastructure.Core/Presentation/Microbenchmarks/Markdown.cs:181
GetValueOrDefault(column)onDictionary<string,double>returns0when the metric is missing, sobaselineValue.HasValueis always true and missing metrics render as 0 (anddeltaPercentcan divide by 0). This can silently produce incorrect tables when a column can’t be computed (e.g., GC-derived columns when traces are unavailable). UseTryGetValueand treat missing metrics as null/empty, and guard the percent calculation when the baseline value is 0.
foreach (var column in configuration.Output.Columns)
{
double? baselineValue = lr.AveragedBaselineOtherMetrics.GetValueOrDefault(column);
double? comparandValue = lr.AveragedComparandOtherMetrics.GetValueOrDefault(column);
string baselineResult = baselineValue.HasValue ? Math.Round(baselineValue.Value, 4).ToString() : string.Empty;
string comparandResult = comparandValue.HasValue ? Math.Round(comparandValue.Value, 4).ToString() : string.Empty;
double? delta = baselineValue.HasValue && comparandValue.HasValue ? comparandValue.Value - baselineValue.Value : null;
string deltaResult = delta.HasValue ? Math.Round(delta.Value, 4).ToString() : string.Empty;
double? deltaPercent = delta.HasValue ? (delta / baselineValue.Value) * 100 : null;
string deltaPercentResult = deltaPercent.HasValue ? Math.Round(deltaPercent.Value, 4).ToString() : string.Empty;
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 22 out of 22 changed files in this pull request and generated 6 comments.
Comments suppressed due to low confidence (1)
src/benchmarks/gc/GC.Infrastructure/GC.Infrastructure.Core/Analysis/BdnJsonResult.cs:25
- The BDN JSON POCOs declare many non-nullable reference-type properties (e.g.,
HostEnvironmentInfo.BenchmarkDotNetCaption,Benchmark.Measurements,BdnJsonResult.Benchmarks) without initialization. With<Nullable>enable</Nullable>, this produces CS8618 warnings and also makes it easy for deserialization to yield nulls at runtime if fields are absent. Consider making these properties nullable where appropriate or initializing with= null!;/ empty collections for required fields.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
This PR aims at calculating average value of multiple microbenchmarks results. The work revolves around: